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Abstract 


The efficacy of defense acquisition is highly dependent upon acquisition 
workforce (AWF) quality, but assessing such quality remains a major challenge, 
particularly given the knowledge-intensive and dynamic nature of acquisition 
organizations and processes. Hence, it is difficult to gauge—much less predict—the 
impact of leadership interventions in terms of policy, process, regulation, 
organization, education, training, or like approaches. Building upon the development 
and application of Knowledge Flow Theory over the past couple of decades, we 
have developed a state-of-the-art approach that enables us analyze, visualize, and 
measure dynamic knowledge and performance. The main idea is to apply this 
approach inwardly to measure the dynamic knowledge and performance of 
acquisition processes (e.g., within contracting and project management 
organizations), but we also look outwardly (e.g., at warfare processes at the tactical 
edges of military combat organizations) to conceptualize an operational proxy for 
acquisition workforce quality: end customer performance. This proxy offers its best 
potential to complement, not replace, other metrics in use, development, and 
conceptualization today, but it arguably concentrates on one of the most important 
AWF quality determinants: how acquired systems affect operational performance. 

Keywords: workforce quality, leadership interventions, Knowledge Flow 
Theory, end customer performance, revolutionary dynamic knowledge, and 
performance measures 
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I 


Introduction 


Acquisition is big business. The U.S. Department of Defense (DoD) alone 
routinely executes 12-figure budgets for research, development, procurement, and 
support of weapon systems and other military products and services (Dillard & 
Nissen, 2005). Acquisition is also a knowledge-intensive business. In addition to 
myriad laws governing federal acquisition in the U.S., a plethora of rules and 
regulations specify—often in great detail—how to accomplish the planning, review, 
execution, and oversight of defense acquisition programs, large and small, sole- 
source and competitive, military and commercial (Dillard, 2003). 

As a result in part—and due to high complexity, multiple stakeholders, goal 
incongruence, open process execution, and large pecuniary rewards for some 
participants—acquisition has been a problematic business too. Seemingly every 
decade, acquisition problems must be addressed by another Blue Ribbon panel and 
reformed yet again. The Better Buying Power Initiatives (BBPI), as a recent instance, 
mandate efficiency and productivity improvements in five acquisition business areas: 
(1) affordability and cost growth, (2) productivity and innovation in industry, (3) 
competition, (4) tradecraft in services acquisition, and (5) non-productive processes 
and bureaucracy (Office of the Under Secretary of Defense for Acquisition, 
Technology, and Logistics [OUSD(AT&L)], 2010). These initiatives focus principally 
on incentives for and interactions with contractors. The Defense Acquisition 
Workforce Improvement Act (DAWIA), as another instance, was signed into law in 
1990 and emphasizes the education, training, and certification of people in the 
acquisition workforce (AWF). Of course, the two leadership interventions are related: 
people in the AWF need to know how to effect the kinds of efficiency and 
productivity improvements mandated via the BBPI. 

These characteristics of acquisition emphasize the criticality of quality in the 
AWF itself: with so much at stake, and in such a knowledge-intensive environment, a 
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high-quality workforce is essential to competent and professional acquisition 
performance. 

These characteristics also elucidate the central role played by people and 
organizations in the AWF: people must be knowledgeable and work effectively—not 
only in terms of their own professional acquisition activities but also with many 
others in acquisition and customer organizations—in order to accomplish key 
objectives and ensure timely, affordable, and responsive delivery of products and 
services to fighting and support units, at home and abroad. Indeed, we understand 
well how the efficacy of defense acquisition is inextricably dependent upon 
workforce quality. Hence, leadership interventions along these lines appear to be 
highly appropriate and on target. 

Assessing the impact of interventions such as these is a challenge, however 
(Assistant Secretary of the Navy for Research, Development, and Acquisition 
[ASN(RDA)], 2011a, 2011b). It’s unclear whether the relatively recent BBPI, for 
instance, have had sufficient time to produce measurable impact. Even after two 
decades of the DAWIA, as another instance, efficacy remains challenging to assess, 
for many extant measures (e.g., number of Defense Acquisition University 
graduates, procurement lead times, program cost growth) fail to account for critical 
aspects of the AWF and important impacts on acquisition performance. Indeed, it is 
difficult to gauge—much less predict—the impact of any leadership interventions 
along these lines (e.g., how much better the AWF has become, or even if it is 
improving over time). Hence, the impact of any particular leadership intervention is 
left largely to anecdote and optimism. To help trim acquisition budgets and guide 
leadership, an improvement in assessing leadership initiatives and interventions is 
needed. 

Since acquisition is a knowledge-intensive endeavor (Snider & Nissen, 2003), 
the knowledge stocks of people comprising the AWF represent likely indicators of 
quality (e.g., education levels, training courses, years of experience, certification 
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levels). However, such indicators are relatively static, pertaining to levels of 
knowledge that change comparatively slowly (Nissen, 2006a). In contrast, 
acquisition laws, rules, and regulations are revised frequently, and acquisition 
knowledge can change abruptly and render obsolete even huge stocks over time. 
Indeed, this dynamic acquisition environment requires members of the AWF to 
sustain career-long learning and knowledge development just to remain proficient as 
acquisition professionals. Thus, as indicators of AWF quality, static knowledge 
stocks appear to be out of phase with the highly dynamic nature of the acquisition 
environment. 

Moreover, acquisition organizations experience persistent flux (Snider & 
Nissen, 2003). We understand well that no two acquisition projects, programs, 
organizations, customers, or requirements are completely alike. Hence, even well- 
educated and -trained people, with appropriate certification levels and years or 
decades of acquisition experience, must continually learn afresh and expand their 
knowledge further with each new assignment. Likewise, it is clear that most 
acquisition organizations form and reform with new people (e.g., via personnel 
transfer, turnover, retirement, promotion) continuously and that end customer needs 
shift perennially (especially at the tactical edges of warfare organizations). Due to 
such discontinuous membership (Ibrahim & Nissen, 2007), even these educated, 
trained, certified, and experienced people must learn repeatedly to trust and work 
effectively with many others—each time someone new joins or leaves a particular 
acquisition organization, and each time a novel product, service or customer is 
involved. Thus, dynamic knowledge also appears to be an important AWF quality 
indicator. 

Further, the pace of change in both information technologies and military 
operations causes this importance of dynamic knowledge to apply in particular 
where information systems (IS) are acquired to support people at the tactical edges 
of warfare organizations. Not only must acquisition personnel be competent in their 
professions—including the acquisition and maintenance of new acquisition 
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knowledge and skills—and continually learn afresh amidst constant organizational 
flux, but they must also keep pace with incessant technological change and satisfy 
customers’ dynamic needs. Even highly competent professionals executing internal 
acquisition processes perfectly can fail to satisfy end customers’ materiel or service 
needs. This presents a huge challenge in terms of assessing AWF quality. 

Building upon the development and application of Knowledge Flow Theory 
(KFT) over the past couple of decades (e.g., see Nissen, 2006b), we have 
developed a state-of-the-art approach that enables us to analyze, visualize, and 
measure dynamic knowledge and performance. This measurement-based approach 
offers potential to overcome the limitations of static measures, as previously 
summarized, by focusing inwardly on the dynamics of knowledge important to 
professional and effective acquisition performance. The main idea is to measure the 
dynamic knowledge and performance of acquisition processes (e.g., within 
contracting and project management organizations). This would represent a 
substantial step forward in terms of acquisition research. 

Further, leveraging complementary research in command and control (C2; 
Nissen & Gallup, 2012), we see potential to use this same measurement-based 
approach to also look outwardly at the dynamics of knowledge important to 
professional and effective warfare performance. Although the specific kinds of 
knowledge required for effective warfare will clearly differ from those essential for 
proficient acquisition, the approach is similar. The main idea is to measure the 
dynamic knowledge and performance of warfare processes (e.g., at the tactical 
edges of military combat organizations). This would represent a substantial step 
forward in terms of C2 research. 

Moreover, we seek to link these inward and outward focusing approaches to 
conceptualize an operational proxy for AWF quality: end customer performance. In 
addition to measuring the dynamic knowledge and performance of key people and 
organizations associated with IS acquisition, for instance, we wish to assess AWF 
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quality by also measuring the dynamic knowledge and performance of primary 
beneficiaries of such systems acquisition: end customers operating at the tactical 
edges of warfare organizations. This proxy offers its best potential to complement, 
not replace, other metrics in conceptualization, development, and use today, but it 
arguably concentrates on one of the most important AWF quality determinants: how 
acquired systems affect operational performance. Two fundamental research 
questions follow accordingly: 

• How can dynamic knowledge and performance metrics be applied to 
assess acquisition workforce quality? 

• How can dynamic knowledge and performance metrics be extended to 
the tactical edges of warfare organizations? 

In this exploratory study, we examine fast-changing IS acquisition from the 
perspective of warfare at the tactical edge, and we consider dynamic knowledge and 
performance measures to both complement and contrast with extant, engineering- 
oriented metrics used to specify and assess most acquired systems today. We begin 
with a summary of KFT and measurement and then follow with the research method 
guiding the study. Preliminary results follow and suggest considerable promise, 
particularly where acquisition personnel and organizations can learn and track how 
changing system characteristics correspond with operational performance at the 
tactical edges of warfare organizations overtime. This technical report concludes 
with key observations, limitations, and an agenda for continued research along these 
lines. 
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II 


Knowledge Flow Theory and Measurement 


The dynamic nature of knowledge indicates that both stocks and flows are 
important (Direickx & Cool, 1989). Knowledge stocks have been comparatively 
straightforward to measure historically; metrics pertaining to education levels, 
training courses, years of experience, certifications, and like knowledge-oriented 
factors are employed broadly. Alternatively, knowledge flows have been 
comparatively much more difficult to assess; metrics pertaining to dynamic 
knowledge—particularly at the group and organization levels—are more elusive. The 
development and application of KFT (e.g., see Nissen, 2006b) over the past couple 
of decades has augmented the set of tools and techniques available to analyze, 
visualize, and measure dynamic knowledge and performance in the organization. 

KFT is founded on a set of 30 principles that characterize dynamic 
knowledge. Such principles are actionable and empirical, and they support the 
diagnosis of workflow and knowledge-flow process pathologies, visualization of 
improvement interventions, and measurement of dynamic knowledge and 
performance gains (Nissen, 2006a). Dynamic knowledge is delineated via five¬ 
dimensional (5D) vector space. Knowledge-flow vectors carry measurements and 
elucidate diagnostic inferences pertaining to the people, processes, and 
organizations associated with knowledge work. Figure 1 illustrates the idea. 

Briefly, the vertical axis “Explicitness” characterizes the nature of knowledge 
along a tacit-explicit continuum. Tacit knowledge implies understanding and know¬ 
how/why, and it is associated most closely with the experiences of people (e.g., 
stemming from job assignments, mentoring, and teamwork) and routines of 
organizations (e.g., including culture, process, and ritual). Explicit knowledge implies 
awareness and know-who/what/where/when, and it is associated most closely with 
artifacts (e.g., documents, formulae, software). Generally, the more tacit the 
knowledge, the greater its appropriability and potential impact on positive 
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performance becomes (Saviotti, 1998). One can measure knowledge explicitness 
using ordinal, interval, or ratio scales. 



Figure 1. 5D Knowledge Flow Diagram 

The horizontal axis “Reach” characterizes how broadly knowledge is known 
and shared in an organization. Here we operationalize reach in terms of the number 
of people in an organization who have access to and can employ any particular 
chunk of knowledge, but we could view reach in terms of organizational levels 
instead (e.g., individual, group, organization, interorganization). Generally, the 
broader the reach of knowledge, the greater its amplification and potential impact on 
positive performance becomes (Nonaka, 1994). Measurements can be made using 
ordinal, interval, or ratio scales. 

The axis “Life cycle” characterizes what is being done with a particular chunk 
of knowledge at some specific point in time. Here we include three activities: (1) 
some individual in the organization learns or creates new knowledge; (2) he or she 
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shares existing knowledge with or transfers it to other people in the organization; 
and (3) one or more people in the organization use or apply existing knowledge to 
accomplish work. Generally, knowledge does not become useful until it is used or 
applied (Pfeffer & Sutton, 1999). Measurements can be made using categorical or 
ordinal scales. 

Because visualization beyond three dimensions is difficult, we represent the 
dimension “Flow time” in terms of the thickness of lines used to delineate vectors. As 
shown in the key to the right of Figure 1, relatively thin lines are used to delineate 
short and fast knowledge flows, whereas comparatively thick lines represent 
knowledge that takes a long time and flows slowly. Generally, the more quickly that 
knowledge flows (e.g., across people, organizations, places, times), the greater its 
potential impact on positive performance becomes (Nissen, 2002). Measurements 
can be made using ordinal, interval, or ratio scales. 

The dimension “Power” is represented similarly in terms of line style used to 
delineate knowledge-flow vectors. Knowledge that flows with relatively low power— 
this corresponds with relatively low performance levels of organizational activities 
enabled by the knowledge—is delineated through orange, dotted lines, whereas 
knowledge flows exhibiting high power—and hence enabling high performance—are 
delineated via purple, solid lines. Measurements can be made using ordinal, interval, 
or ratio scales. 

Integrating these five dimensions graphically and analytically generates a 5D 
vector space to examine dynamic knowledge. Such 5D space and examination 
schemes are completely general: they can be applied to any dynamic knowledge in 
any organizational domain (e.g., acquisition, command and control, software 
engineering). 

As an example of use and application, consider Figure 2, which illustrates an 
important knowledge flow desired by the organization. Point A represents one 
individual in the organization who learns something new (to that organization) or 
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creates entirely new knowledge. In terms of the 5D space, this represents tacit 
knowledge that is created by an individual (i.e., one person); hence, its position at 
the bottom-back corner of the diagram. 



Figure 2. Knowledge Creation and Application Needs 

In the acquisition domain, for instance, consider that such new knowledge 
could pertain to a technique for reducing the acquisition time for an important IS 
needed in the field. Because information technology (IT) advances so quickly— 
outpacing the ability of many acquisition organizations to develop and field systems 
responsively—the organization views this new knowledge created at Point A as 
important, and it would like to see such knowledge shared with and applied by all 
100 people in that organization who work with IT. 

Such application by 100 people in the organization is represented by Point B. 
The thin, purple, solid vector connecting Points A and B represents the desired 
knowledge flow: the organization wishes for such knowledge to flow quickly and with 
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high power (e.g., enabling all 100 people at Point B to work, within one day, at the 
same performance level as the innovative individual at Point A). This represents a 
5D knowledge flow vector. A question mark in the figure next to the vector indicates 
that such a fast, powerful knowledge flow is desired by the organization, but it is 
unclear which, if any, organizational process can enable it. 

This leads to Figure 3, which depicts a ridge, or obstruction, that prevents 
knowledge from flowing quickly and powerfully from Points A to B as desired by the 
organization. Practically, the organization lacks a process for such quick and 
powerful knowledge to flow directly as delineated in Figure 2. Indeed, most 
organizations do lack such a process (Nissen, 2006a). Some other approach to 
sharing and applying the important IT acquisition knowledge is required. 



Figure 3. Knowledge Flow Obstruction 


Figure 4 delineates two, alternate, archetypical knowledge flows 
corresponding to processes that are within this organization’s capabilities. (We say 
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archetypical, because most organizations employ these classic processes routinely, 
and because they present a vivid contrast in terms of how dynamic knowledge 
flows.) One knowledge flow is depicted in terms of a relatively fast (i.e., thin lines) 
but low-power (i.e., orange, dotted lines) vector series; this first flow is associated 
with explicit knowledge and utilizes one or more IS for knowledge articulation and 
distribution in explicit form. The other is delineated via a comparatively slow (i.e., 
thick lines) but high-power (i.e., purple, solid lines) vector; this second flow is 
associated with tacit knowledge and utilizes one or more human-centered 
approaches to knowledge sharing (e.g., group interaction, mentoring, personnel 
transfer). 



Figure 4. Alternate Archetypical Knowledge Flows 


In some greater detail, the first knowledge flow consists of three vectors. The 
first vector is represented by a vertical line arising from Point A. This vector depicts 
the individual at Point A articulating his or her new, tacit knowledge via an IS so that 
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it can be shared electronically. Such articulation (e.g., consider writing a procedure, 
developing a training course, posting to an intranet or social networking site) tends 
to be somewhat time-consuming, hence the relatively thick line. Articulating 
knowledge in explicit form also tends to dilute the knowledge in terms of power. 
Reading a book, for instance, about how to accomplish important acquisition tasks 
(e.g., contract negotiation, risk assessment, balancing program cost and schedule 
with performance) is not the same as having direct personal experience 
accomplishing those tasks; hence the orange, dotted line. 

Once articulated in explicit form, however—particularly via IS—the knowledge 
can be shared very broadly (e.g., organization-wide) and very quickly (e.g., within 
seconds), albeit with diluted power; hence the thin, orange, dotted line at the top of 
the diagram. Indeed, one could consider this broad and fast flow as additive to the 
organization’s express acquisition body of knowledge (BOK), which we note at the 
top-right of Figure 4. Such explicit BOK can then be accessed quickly and applied in 
turn by all 100 people in the organization. This articulated, explicit knowledge 
remains relatively diluted and less powerful, nonetheless, so application at Point B 
would not support the same performance level as at Point A; hence the thin, orange, 
dotted line descending down to Point B. 

Alternatively, the second knowledge flow consists of a single vector, although 
it curves and bends through the tacit knowledge plane at the bottom of Figure 4. 

This vector depicts the individual at Point A applying his or her new, tacit knowledge 
and then sharing it with some number of other people (say, 10 people, as illustrated 
in Figure 4) through one or more techniques such as extended group interaction, 
mentoring, or personnel transfer to work directly with different coworkers across the 
organization. 

Once each of these ten people has learned the new, tacit knowledge, then all 
of them can continue the process and share it using similar techniques (e.g., group 
interaction, mentoring, or personnel transfer) with others. Through such a process, 



Acquisition Research Program 

GRADUATE SCHOOL OF BUSINESS & PUBLIC POLICY 
NAVAL POSTGRADUATE SCHOOL 


- 13 - 




100 people (i.e., 10 people each sharing with another 10 people) can learn this new, 
tacit knowledge to the extent necessary for powerful application at Point B. This 
knowledge flow is depicted by a thick vector to indicate that it occurs comparatively 
slowly, but such vector is also delineated by a purple, solid line to show that the 
corresponding knowledge has high power and enables knowledge-based action at 
the same performance level as the individual who created it at Point A. 

The key is, one can measure these five dimensions of knowledge—whether 
via explicit or tacit flows—and relate them to the corresponding knowledge-based 
process performance by people in the organization. Indeed, by correlating such 
dynamic knowledge measures with performance metrics, one can develop a model 
capable of analyzing, visualizing, and even predicting process performance based 
upon knowledge flow patterns. 

Of course, many diverse combinations of these archetypical knowledge flows 
are possible too, yet most knowledge flows are likely to reflect some aspects of 
these two dynamic patterns (Nissen, 2006b). Through empirical analysis and 
calibration of specific knowledge flowing through any particular organization in the 
field, one can correlate 5D dynamic knowledge flows with work performance, 
resulting in a model capable of measurement and prediction. Through this 
technique, we are working to assess AWF quality in terms of dynamic knowledge 
flows. 
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Ill 


Research Method 


The first research question, articulated previously, includes a “how” 
interrogative and suggests that a qualitative method may be most appropriate to 
investigate it (Yin, 1994). Despite the generality of KFT and the 5D space described 
in the previous section, applying the corresponding analytic, visualization, and 
measurement techniques to assess AWF quality requires acquisition domain 
knowledge in general and process-specific understanding in particular. We need to 
study one or more specific acquisition processes in detail in order to apply the 
techniques and assess workforce quality. The case study method is highly 
appropriate for an investigation along these lines (Benbasat, Goldstein, & Mead, 
1987; Eisenhardt & Graebner, 2007; Yin, 1994), and we conduct just such a case 
study in parallel with the investigation reported here. 

The second research question, stated previously, also involves a “how” 
interrogative, and it likewise suggests a qualitative method. However, this second 
question calls for an extension of dynamic knowledge and performance 
measurement out to the tactical edges of warfare organizations, and hence is much 
more exploratory from an acquisition perspective. Because we seek an operational 
proxy for AWF quality, we investigate dynamic knowledge and performance through 
explicit examination of three warfare organizations and processes that are far 
removed from core acquisition. 

One organization operates within a U.S. Navy fleet and has units deploying 
rhythmically to war zones and other areas overseas. A second organization operates 
within a Navy systems command but concentrates on ensuring the readiness of this 
same fleet. The third organization permeates functionally throughout naval 
operations and is responsible for information dominance. By interacting with 
knowledgeable representatives from each of these three organizations—and it is 
very important to note that these are warriors and other operational personnel, not 
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acquisition professionals—we gain considerable insight into the key knowledge 
dynamics associated with warfare at the tactical edges. 

Further, by triangulating between these three organizations, we identify a 
critical, knowledge-intensive process that can be represented with sufficient fidelity 
and granularity to suggest feasible application of our dynamic knowledge and 
performance measures. The process has the somewhat unwieldy name Tasking, 
Collection, Processing, Exploitation, and Dissemination, to which we refer simply by 
its acronym TCPED. In the results that follow, we delineate the TCPED process and 
seek to apply our dynamic knowledge and performance measures to it. We then 
attempt to interpret such application and to elucidate insight into assessing AWF 
quality via proxy. 
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IV 


Results 


Results from this exploratory investigation center on delineating the TCPED 
process, elaborating an insightful subprocess in detail, and applying our dynamic 
knowledge and performance measures to it. We discuss these in turn and then focus 
on elucidating insight into AWF quality. 

A. TCPED 

TCPED does not represent a new operational process per se, but with the 
U.S. Navy’s relatively recent creation of its Information Dominance Corps, it has 
attracted considerable attention as a critical complement to the Find, Fix, Target, 
and Track (F2T2) process associated broadly with combat operations. The key F2T2 
issue remains “knowledge — finding the targets” (Keeter, 2004), and as a 
knowledge-intensive process, TCPED addresses this issue directly, and hence 
represents a promising target of study. 

Given the knowledge-intensive nature of TCPED, its execution is enabled 
fundamentally by IT, and IS are acquired routinely with the goal of enhancing 
warfare efficacy. This nature provides an excellent link back to our fundamental 
research question and interest in the AWF. From the operational perspective of 
TCPED participants at the tactical edges of organizations, IS acquired and fielded to 
enhance warfare efficacy should accomplish just that: enhance warfare efficacy. 
Further, such efficacy enhancement should be measurable. 

The problem is, it is difficult to understand—much less measure—how well 
any particular warfare process is working, which of many different organizational 
arrangements are best across diverse missions, or how well various IS enhance or 
impede the process. Indeed, when seeking to acquire new IT and like technologies 
to enhance warfare efficacy, system implementation can make the operational 
processes worse in the battle space, and it is increasingly common for different 
acquired systems to fail in terms of interoperating (Nissen & Gallup, 2012). 
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Indeed, modern warfare efficacy requires a combination of people and 
technologies to enable warriors to leverage local knowledge and seize emergent 
opportunities to achieve commanders’ intent across distributed organizations. This 
requirement highlights further the critical role played by TCPED, which seeks to 
enable commanders and warriors at the tactical edges to put dynamic knowledge 
into effective action, with or without IS in development or in the field. 

Additionally, unlike many stable, mature, and well-understood warfare 
processes, TCPED remains in a constant state of analysis, refinement, and 
development. Hence, it represents a rapidly moving target for IT development, and 
engineering-oriented metrics used to evaluate most IS fail to address how dynamic 
knowledge translates into effective (or ineffective) action. Moreover, with current 
analytical models and metrics, it remains unclear how to assess whether any 
particular refinement in the warfare process, new IS implementation, or like change 
will lead to increased TCPED efficacy or whether performance will degrade instead. 
This lack of clarity illuminates a capacious gap between the efficiency of IT 
acquisition and the warfare efficacy of IS employment at the tactical edge. 

Given the dynamic nature of the TCPED process, as characterized 
previously, we bound the scope of this exploratory project by concentrating on a 
particularly important and knowledge-centric subprocess: exploitation. Such 
bounding enables us to examine, within a single exploratory study, the feasibility of 
our approach to measuring the dynamic knowledge and performance of this 
operational process performed at the tactical edges of naval organizations. Follow- 
on researchers can then extend these promising results via subsequent studies 
through the process as a whole and, in turn, to other warfare processes seeking to 
benefit from IT acquisition. 

B. TCPED Exploitation 

Figure 5 delineates the principal tasks comprising TCPED exploitation. In this 
figure, process activities are depicted as rectangular boxes connected to one 
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another via arrows to delineate the process workflow. Each process activity is 
situated within a horizontal region (e.g., referred to widely as swim lanes) that 
depicts the responsibility of a particular organizational group to accomplish it. For 
several instances, the leftmost process activities—“Correlate, Fuse Multi-lnt Info” -> 
“Operations Environment Impact” -> “Evaluate Adversary” -> “Develop Adversary 
COA”—are shown connected together as responsibilities of the “Assessor” group; 
the “Develop Adversary COA” activity interrelates with “Watch Analyst Coordination,” 
the latter of which is shown as the responsibility of the Joint Intelligence Operations 
Center (JIOC), and which interrelates in turn with the Joint Operations Center (JOC) 
activity “Watch Coordination.” 



X ^Convert ▼ Eft Select 
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’ I* Pandora Radio... & PYTHON ' Naval Postgra... ■£> Naval Postgra... 0 C:\Mark\re... X | | ^ 0 ’ □ # T Page-r Safety^ Tools-■ 



Figure 5. TCPED Exploitation Process Flow 

Other instances pertain to “Assess Near-Term Ops Impact,” the output of 
which activity provides important knowledge and information to Operations (“J3”); 
“Daily Update Information” and “Propose New Focus Areas,” the output of which 
activities provide important knowledge and information to Intelligence Management; 
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and “Determine Emerging l&W” and “Dissem l&W,” both of which activities are 
performed by and are the responsibility of the Assessor as well. We omit graphical 
depiction or discussion of the other TCPED exploitation activities, because our intent 
is not to be exhaustive here, and these should suffice for our present purposes. 

In particular, discussions with the knowledgeable people interviewed through 
this research indicate that the tasks labeled “Evaluate Adversary,” “Develop 
Adversary COA,” and “Assess Near-Term Ops Impact” are especially important and 
require considerable tacit knowledge. Recall that tacit knowledge, as powerful as it 
is, tends to flow relatively slowly and narrowly through organizations. This makes it 
particularly challenging to support via IT, and it provides an excellent focus for our 
exploration. Indeed, the people performing these activities must develop substantial, 
tacit knowledge pertaining to adversaries’ capabilities, likely actions, and their 
consequences in terms of friendly forces and operations. Such tasks also clearly 
require relevant and timely information, but knowledge of the adversary is key here, 
and the effectiveness of these tasks can contribute greatly to—or, if ineffective, 
impair instead—commanders’ decision-making and warriors’ actions on the tactical 
edge. 


By focusing on how dynamic knowledge flows through warfare process 
activities such as these, and especially by linking the activities to knowledge-based 
actions enabled at the tactical edge, we can examine how well knowledge is flowing 
and supporting tactical action. Specifically, by integrating the organizations, 
personnel, and activities included in the exploitation process diagrammed in Figure 5 
with key dimensions from KFT, we seek to identify critical paths in the process 
where knowledge is flowing well and appropriately, as well as identifying blocked 
paths where it is not, and we strive to use our dynamic knowledge and performance 
metrics to help overcome any disconnects between IT acquisition and warfare 
efficacy. 
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C. Dynamic Knowledge Flows 


Through detailed analysis, we can delineate the principal knowledge flows 
enabling TCPED exploitation. Taking Develop Adversary COA as an express 
example, the people performing this activity rely fundamentally upon experience- 
based tacit knowledge (e.g., military tactics, adversary capabilities, organizational 
vulnerabilities). Although formal training courses, professional educational programs, 
and like approaches contribute to these knowledge flows, such knowledge is 
accumulated principally through direct experience (i.e., on-the-job training [OJT]), 
often over many years or even decades. 



Figure 6. Military Tactics Knowledge Flows 

Figure 6 delineates how military tactics knowledge, for instance, accumulates 
through cyclic iteration between applying one’s existing tacit knowledge (labeled 
“Tactics K application” in the figure) and learning from the resulting experience 
(labeled “Tactics K creation” in the figure). We locate this cyclic knowledge flow 
vector at the individual level of reach, indicating that the Develop Adversary COA 
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activity is conducted in this case by a single individual; were multiple people to 
engage jointly in assessments such as this, we would simply relocate the 
corresponding knowledge flows to the group level, with the same basic pattern 
persisting. 

Consistent with our previous discussion, one can observe from Figure 6 how 
the vector for knowledge application is relatively thin, denoting that the flow is 
correspondingly fast, yet this vector is delineated via a purple, solid arrow, denoting 
that the flow reflects powerful, tacit knowledge; that is, once tacit knowledge has 
been acquired over time, it can be applied relatively quickly. In partial contrast, the 
complementary vector for knowledge creation is comparably thick, denoting that the 
knowledge acquisition flow is relatively slow, yet this vector is also delineated via a 
purple, solid arrow, similarly denoting that the flow reflects powerful, tacit knowledge 

Continuing with the Develop Adversary COA example, the people performing 
this activity also rely on a situated understanding of the organization’s current 
mission-environment context, the adversary evaluation synthesized in the preceding 
exploitation process step, and contemporaneous knowledge regarding both current 
and future operations being conducted and planned, respectively, by the 
organization. Knowledge flowing to enable these process activities follows 
somewhat different patterns than those activities pertaining to military tactics. In 
particular, these latter knowledge flows involve interactions across different 
organizational groups, and they involve both tacit and explicit knowledge. 

For instance, Figure 7 delineates three knowledge flows associated with tacit 
knowledge sharing and intergroup accumulation. The leftmost cyclic vector (labeled 
“Individual K accumulation”) is comparable to that discussed previously in Figure 6, 
except that instead of military tactics knowledge, it pertains to the latter knowledge 
flows (e.g., associated with current mission-environment, adversary evaluation, and 
current and future operations). We continue to focus on individual knowledge 
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accumulated by a single person—in this case, within the assessor group—but notice 
that we include a similar cyclic vector located at the intergroup level. 



Figure 7. Tacit Knowledge Sharing & Intergroup Accumulation 

This latter vector (labeled “Intergroup K accumulation”) reflects tacit 
knowledge accumulating across different organizational groups; multiple individuals 
from a variety of groups work and learn from their experiences together. The 
intergroup vector follows the same cyclic pattern as that seen with individual OJT, 
only at a higher organizational level. As with individual knowledge accumulation, this 
intergroup accumulation is delineated by a cyclic, purple, solid vector reflecting 
knowledge application and creation occurring at two different rates (i.e., quickly and 
slowly, respectively). 

A third vector (labeled “Tacit K sharing”) links the other two. Such tacit 
knowledge sharing reflects individuals—who accumulate knowledge (especially via 
OJT) within their separate groups—sharing knowledge with people in other groups 
through conversation, dialogue, face-to-face (F2F) interaction, and like means. The 
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two-headed arrow included with this sharing vector depicts knowledge flowing bi¬ 
directionally: individuals share knowledge across groups in the organization, and 
they also learn through this knowledge process. 

As with the two cyclic vectors delineated and discussed previously, 
knowledge flows corresponding to such tacit sharing are depicted with a purple, solid 
vector to designate powerful tacit knowledge, and the vector is depicted with a 
relatively thick line to indicate that tacit knowledge flows across organizational 
groups tend to accumulate relatively slowly. However, we depict this sharing vector 
with a line that exhibits intermediate thickness; that is, the vector is thicker than the 
application vector lines—suggesting that tacit knowledge application flows across 
groups (e.g., in a matter of days, weeks, and months) more slowly than via individual 
application (e.g., in a matter of minutes, hours, and days)—but thinner than the 
creation vectors—suggesting that such cross-group knowledge can flow more 
quickly than can individual accumulation of experience-based tacit knowledge (e.g., 
in a matter of months, years, and decades). 



Figure 8. Explicit Organizational Knowledge Sharing 
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As another instance, Figure 8 delineates alternate knowledge flows 
associated with explicit organizational knowledge sharing. The leftmost cyclic vector 
(labeled “Individual K accumulation”) is identical to that discussed previously in 
Figure 7 (e.g., cyclic, purple, solid, powerful, tacit). We continue to focus on 
individual knowledge accumulated by a single person—in this case, within the 
assessor group—but notice that we include a three-segment flow (labeled “Explicit K 
sharing”) to depict knowledge being shared organization-wide in explicit form. 

This three-segment flow begins with a vertical vector rising up out of the tacit 
plane, as an individual (i.e., in the assessor group) articulates his or her tacit 
knowledge into explicit form (e.g., via textual reports, graphical sketches, digital 
images). This articulation can be a time-consuming process; hence, the 
corresponding knowledge flow vector is depicted by a relatively thick line. In 
addition, we understand that such articulated, explicit knowledge does not reflect the 
same power level as the tacit knowledge used for its creation; hence, the 
corresponding knowledge flow vector is depicted by an orange, dotted line. 

The second vector comprising this three segment flow begins where the first 
vector terminates. Once articulated in explicit form, such knowledge can be stored, 
replicated, and disseminated quickly and broadly via one or more IS (e.g., intranet 
document repositories, online sharing tools, common operational displays). This 
second vector in the segment is delineated by a thin line to denote fast knowledge 
flows, but the line remains orange and dotted to depict its diluted power. The third 
vector in the segment is also depicted by a thin, orange, dotted vector, which 
represents this same, diluted, explicit knowledge flowing via IS quickly and broadly 
across the organization. 

D. Dynamic Knowledge and Performance Measurement 

Through detailed analysis, we identify and operationalize three KFT metrics 
that appear to be particularly insightful for our present purposes: knowledge reach 
(i.e., how many people in the organization share specific chunks of knowledge), 
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knowledge flow time (i.e., how long it takes chunks of tacit and explicit knowledge to 
flow from where and when they are to where and when they’re needed), and 
knowledge power ( i.e., the performance level of knowledge-enabled work). 
Continuing with Develop Adversary COA as an express example, we can begin to 
quantify the key knowledge flows delineated previously. 

Table 1. ROOM Knowledge Flow Measurement 


Knowledge Flow 

Reach 

Flow Time 

Power 

Individual K Accumulation 

1 

Years 

Very High 

Intergroup K Accumulation 

10 

Months 

High 

Tacit K Sharing 

10 

Days 

High 

Explicit K Sharing 

100 

Hours 

Diluted 


For instance, Table 1 summarizes rough order of magnitude (ROOM), three- 
dimensional estimates for each of the four knowledge flows delineated and 
discussed previously with respect to the Develop Adversary COA activity within 
TCPED exploitation. In this table, we approximate knowledge flow measurements 
only to an order of magnitude, but we begin to illustrate the use and utility of the 
approach, and we outline a method for obtaining higher fidelity measurements in 
practice. 

In the first column of the table, we list each of the four knowledge flows 
discussed previously, and in the other three columns, we summarize ROOM 
estimates for knowledge reach, flow time, and power. Looking first at individual 
knowledge accumulation, the reach is listed as 1; this reflects our previous 
discussion of knowledge being accumulated iteratively at the individual level, hence 
unitary reach. In the table, flow time is listed in order of magnitude as “years” for 
comparison with the other knowledge flows; this reflects our discussion about how 
deep, experience-based tacit knowledge (e.g., pertaining to military tactics) can 
require years or decades to accumulate. Power is listed likewise in order of 
magnitude as “very high” for similar comparison with the other knowledge flows; this 
estimate is somewhat definitional, but it reflects that experience-based tacit 
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knowledge does not suffer from power dilution, and it is meant to reflect the 
considerable power of tacit knowledge accumulated over long periods of time and 
through abundant experience. 

Looking next at intergroup knowledge accumulation, rough estimates for this 
knowledge flow indicate that 10 people can be reached by it; this is an order of 
magnitude larger than that shown for individual knowledge accumulation, and it 
reflects knowledge flowing to multiple people across organizational groups. The flow 
time estimated for intergroup knowledge flows is summarized as “months,” which is 
an order of magnitude faster than that for individual knowledge accumulation; this 
reflects the comparatively lower level of deep knowledge associated with intergroup 
knowledge and work flows, as people across groups interact principally via their 
present assignments—which, in this naval context, generally span less than a year. 
As discussed previously, the power level is listed simply as “high” to reflect that 
intergroup tacit knowledge (e.g., people learning to work well together across 
groups) does not suffer from power dilution, but it also reflects that the power level is 
not comparable to that associated with deep, experience-based knowledge 
accumulated over years of individual experience (e.g., pertaining to military tactics). 

Estimates for the third knowledge flow (i.e., tacit knowledge sharing) are 
similar in terms of reach (10), but they reflect more than another order of magnitude 
reduction in flow time (i.e., “days”); this corresponds to the principle that knowledge 
sharing can be accomplished more quickly than the associated knowledge 
accumulation (Nissen, 2006b). The (“high”) power level matches that for intergroup 
accumulation mentioned previously and for the same reasons. 

In considerable contrast, the flows associated with the fourth knowledge flow 
(i.e., explicit knowledge sharing) are quantitatively very different. We estimate the 
reach at 100 in the table, but the knowledge flows are constrained only by the reach 
of the network infrastructure; hence, this figure could be many orders of magnitude 
larger (e.g., consider a report, through which everyone in a 100,000 person 
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organization has access to the same explicit knowledge). The estimate for flow time 
is similar in that we list it as “hours” (e.g., principally to account for the time required 
to articulate knowledge in explicit form), whereas once made explicit, such 
knowledge can be shared in seconds. 

Moreover, the power level (“diluted”) for this explicit knowledge flow is 
qualitatively different than that for its tacit counterparts; this is also somewhat 
definitional, but it indicates that most people reading written documents, for example, 
will not be expected to perform knowledge-based activities at the same level as the 
people writing those documents. 

E. System Assessment 

The remaining measurement of knowledge power is linked directly to 
performance of the work activities enabled by such knowledge. In the case of 
Develop Adversary COA, to continue our previous example, we could approach 
such measurement via multiple operationalizations. For several instances, we could 
track how much time is required to develop a set of adversary COAs sufficiently well 
for inclusion in a morning flag brief (i.e., appropriate for presentation to a flag officer); 
using the same flag brief criterion, we could count how many sufficiently credible 
adversary COAs are developed within a set time frame (e.g., one day, week, or 
month); we could ask the flag officer and staff in question (e.g., including the Chief of 
Staff and other directly reporting officers) to evaluate the quality of each adversary 
COA presented (e.g., based on criteria of importance to them); or we could pursue 
the development of other, likewise understandable and relevant performance 
measures. Any such performance measure can serve as a quantitative (and possibly 
multidimensional) proxy for knowledge power. 

With one or more such measures in hand, we could then establish a 
baseline—comprised of quantitative measurements for reach, flow time, and 
knowledge power/performance —for the organization as it operates as usual. To 
evaluate some particular IS, we could simply compare this baseline with 



Acquisition Research Program 

GRADUATE SCHOOL OF BUSINESS & PUBLIC POLICY 
NAVAL POSTGRADUATE SCHOOL 


-28 - 






measurements taken as the organization uses the IS under controlled, or at least 
comparable, conditions. For instance, say that we wish to test a prototype IS 
designed to improve tacit knowledge sharing through introduction of social media 
techniques; we could measure the knowledge flows both with and without such IS to 
assess its impacts. 

Specifically, using one or more proxy measures as suggested previously 
(e.g., time required to develop a set of adversary COAs for a flag brief, how many 
adversary COAs are developed, flag officer quality evaluation, others), we could 
conduct an experiment in the laboratory or in the “field” (e.g., on deployed ships at 
sea) and measure knowledge and performance directly. As an experiment to 
compare performance with and without the prototype IS, for instance, we would 
ideally like to see the same people, performing the same tasks, in the same 
environments and settings, at the same times of day, seasons of year, weather 
conditions, sea states, and other factors to isolate use of that IS as the only 
difference. In other words, one set of dynamic knowledge measurements would be 
taken for performance in the baseline situation; a second set of measurements 
would be taken for performance with a prototype IS; and, ideally (e.g., with good 
experiment design and techniques), the difference would represent solely the effect 
of that IS. 

With these measurements in hand, the difference in task performance 
becomes an operational measure of IS efficacy; that is, if the only difference 
between experiment cases is whether the prototype IS is used or not, and task 
performance is measurably better or worse in one case or the other, then we have a 
knowledge-based assessment of how well such IS improves (or worsens) work 
performance at the tactical edge of the organization (e.g., TCPED exploitation). 
Moreover, in addition to using traditional, engineering-oriented performance 
measures (e.g., bandwidth, technical reliability, memory), this assessment can be 
employed to evaluate the IS operationally—and under controlled conditions—not just 
technically. The potential is huge. 
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Further, given sufficient experience with conducting experiments along these 
lines, this approach can even be used to specify new IT and other systems to be 
acquired; that is, in conjunction with using only engineering measures of IS 
performance, for instance, the acquisition organization can specify improvement in 
operational task performance as a key criterion for evaluation. This way, acquisition 
personnel can conduct efficient system acquisitions, and warriors on the tactical 
edges of organizations can use systems that improve their work performance. We 
bridge the gap between acquirer and warrior, and everybody wins. 

F. Illustrative Example 

In this section, we include an illustrative example of application to an 
hypothetical IT system competition. We use only representative values for illustration 
here, but the approach and associated techniques can be applied directly to system 
competitions in the field. For continuity, we continue with the Develop Adversary 
COA task discussed previously, and we build upon the rough knowledge flows and 
measurements reported previously. 


Table 2. Baseline Knowledge Flow Measurement 


Knowledge Flow 

Reach 

Flow Time 

Power 

X-Power 

Tacit K Sharing 

10 

20 Hours 

95% 

9.5 

Explicit K Sharing 

100 

2 Hours 

5% 

5.0 


Table 2 recapitulates the most relevant measurements reported in Table 1 for 
what we term the baseline, representing the Develop Adversary COA task as it is 
performed today (i.e., sans new IS); that is, the baseline measurements are used for 
comparison with this same task performed with the support of two competing IS 
prototypes: (1) a social media application designed to improve tacit knowledge 
sharing, versus (2) a document collaboration application designed to improve explicit 
knowledge sharing. 

Notice in Table 2 that we limit our summary to the pair of knowledge flows 
associated directly with the alternate IS: tacit knowledge sharing (addressed by IS-1) 
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and explicit knowledge sharing (addressed by IS-2). Recall, from our discussion 
above, that the knowledge flow corresponding to tacit knowledge sharing reflects 
individuals—who accumulate knowledge (especially via OJT) within their separate 
groups—sharing knowledge with people in other groups through conversation, 
dialogue, F2F interaction, and like means. The central idea of IS-1 is to enable such 
knowledge sharing remotely; that is, the IS intends to enable and promote tacit 
knowledge sharing without the need for (as much) F2F interaction. 

Recall, further from our discussion above, that the knowledge flow 
corresponding to explicit knowledge sharing reflects organizational artifacts (e.g., 
textual reports, graphical sketches, digital images, and like media) that are stored, 
replicated, and disseminated quickly and broadly via intranet document repositories, 
online sharing applications, common operational displays, and like tools. The central 
idea of IS-2 is to enable recipients of assessor reports (e.g., in the JIOC and JOC 
groups) to interact with assessors during report development; that is, the IS intends 
to enhance and accelerate explicit knowledge sharing by providing recipients with 
access to assessor draft reports and to enable communication before finished 
reports are released officially. 

Notice also that we replace the ROOM estimates from Table 1 with 
quantitative values. For instance, the “days” flow time estimate from above for the 
tacit knowledge sharing flow reads as “20 hours” in Table 2. Based on observation 
and discussion, roughly 20 hours are required for key tacit knowledge to complete its 
flows. Further, the “high” power estimate from above reads as “95%” here. As such, 
10 different people outside the assessor group (e.g., in the JIOC and JOC) are able 
to explain the details of each adversary COA from memory with 95% accuracy on 
average; the other way to look at this is that 19 of 20 people can explain the details 
with 100% accuracy. 

Similarly, the “hours” flow time estimate from above for the explicit knowledge 
sharing flow reads as “2 hours” here. This indicates that roughly 2 hours are required 
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for a high-quality and credible adversary COA to be articulated, shared with, and 
understood by recipients. Further, the “diluted” power estimate from above reads as 
5% here. As such, 100 different people outside the assessor group (e.g., in the JIOC 
and JOC) are able to explain the details of each adversary COA from memory with 
5% accuracy on average; the other way to look at this is that five of 100 people can 
explain the details with 100% accuracy. 

Notice finally that we include a fifth column in Table 2 (labeled “X-Power”) to 
represent the induced dimension extended knowledge power. Extended knowledge 
power is calculated as the product of knowledge reach and power levels; it reflects 
the combined distribution and efficacy of knowledge flows. For instance, the 
extended knowledge power for the tacit knowledge sharing flow is shown in the 
Table as 9.5 (i.e., reach of 10 [times] power of .95 = x-power of 9.5), whereas the 
value calculated for explicit knowledge sharing flow is shown as 5.0 (i.e., reach of 
100 [times] power of .05 = x-power of 5.0). 

This respective induction and quantification of the extended knowledge power 
dimension and measure provide us with a technique for comparing the efficacy of 
tacit and explicit knowledge flows directly, despite the significant differences 
between their dynamic characteristics and behaviors (e.g., quick, broad, diluted 
explicit flows versus slow, narrow, powerful tacit flows). Clearly, higher values are 
preferred over lower ones, but organizations face trade-offs regarding whether to 
emphasize explicit or tacit knowledge flows. 

Table 3. IS Supported Knowledge Flow Measurement 


Knowledge Flow 

Reach 

Flow Time 

Power 

X-Power 

Tacit K Sharing (IS-1) 

20 

20 Hours 

75% 

15.0 

Explicit K Sharing (IS-2) 

20 

2 Hours 

10% 

2.0 


For further illustration, Table 3 summarizes these same knowledge flow 
measurements—for the same people, organizations, tasks, and time frames—after 
the prototype IS have been implemented and trained with. This point is important; 
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one cannot expect a new IS to be used effectively and productively before its users 
have been trained adequately. (It’s humorous, nonetheless, how often one sees 
comparisons made without adequate training, particularly in field experiments.) 

In the case of tacit knowledge sharing supported by IS-1, say that the social 
media application enables twice as many people to participate in the conversations 
(i.e., reach extends to 20) within the same 20-hour time frame (e.g., by obviating the 
need for collocation), but the power level decreases to 75% (e.g., due to losses via 
mobile social media applications). Despite the drop in power, the extended reach 
would more than make up for the loss, because of the extended power increase to 
15.0. Alternatively, in the case of explicit knowledge sharing supported by IS-2, say 
that the document-sharing application reduces to 20 the number of people who can 
participate effectively within the same two-hour time frame (e.g., due to interference 
by multiple people interacting with the same documents), yet the power level of 
those who do participate increases to 10% (e.g., stemming from increased textual 
interaction across organizational groups). Despite the increase in power, the 
reduced reach would more than offset the gain, because of the extended power 
decrease to 2.0. 


Table 4. Comparative Knowledge Flow Measurement 


Knowledge Flow 

Baseline 

(X-Power) 

IS Enabled 

(X-Power) 

Difference 

(X-Power) 

Difference 

(Percentage) 

Tacit K Sharing 

9.5 

15.0 

+ 5.5 

+ 58% 

Explicit K Sharing 

5.0 

2.0 

-3.0 

- 60% 


In Table 4, we summarize the comparative results via four measurements. 
First, the Baseline X-Power contrast between the tacit and explicit knowledge 
sharing processes reflects our result from Table 2 (i.e., 9.5 versus 5.0, respectively). 
Second, the IS Enabled X-Power contrast between these same processes reflects 
similarly our result from Table 3 (i.e., 15.0 versus 2.0, respectively). Third, the 
Difference X-Power contrast measures the effect of incorporating the two IS. For 
instance, using IS-1 to support tacit knowledge sharing increases extended 
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knowledge power by 5.5 (i.e., 15.0 - 9.5 = +5.5) for a 58% gain. In contrast, using 
IS-2 to support explicit knowledge sharing decreases extended power by 3.0 (i.e., 

2.0 - 5.0 = -3.0) for a 60% loss. 

Recall that the knowledge power measurement relates directly to 
organizational performance at the tactical edge, on the Develop Adversary COA task 
in this illustrative case. In addition to providing an objective and quantitative 
approach to assessing the potential value (or harm) of an IS of interest, the 
technique described in this report also suggests a way to specify performance 
requirements for candidate IS of interest. 

Consider, for instance, if—in addition to whatever engineering specifications 
are desirable or customary—the specification read along the lines of, “the IS must 
demonstrate at least a 25% increase in X-Power measured during a fleet 
experimentation exercise.” This specification would arguably place considerable 
contractor emphasis on improving knowledge flow and work performance of users at 
the tactical edges of the warfare organizations targeted for the acquisition and 
implementation of their IS. It would also appear likely to help bridge the gap between 
acquisition efficiency and warfare efficacy. 
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V 


Conclusion 


The efficacy of defense acquisition is highly dependent upon acquisition 
workforce (AWF) quality, but assessing such quality remains a major challenge, 
particularly given the knowledge-intensive and dynamic nature of acquisition 
organizations and processes. Hence, it is difficult to gauge—much less predict—the 
impact of leadership interventions in terms of policy, process, regulation, 
organization, education, training, or like approaches. 

Building upon the development and application of Knowledge Flow Theory 
over the past couple of decades, we have developed a state-of-the-art approach that 
enables us to analyze, visualize, and measure dynamic knowledge and 
performance. The main idea is to apply this approach inwardly to measure the 
dynamic knowledge and performance of acquisition processes (e.g., within 
contracting and project management organizations), but we also look outwardly 
(e.g., at warfare processes at the tactical edges of military combat organizations) to 
conceptualize an operational proxy for acquisition workforce quality: end customer 
performance. This proxy offers its best potential to complement, not replace, other 
metrics in use, development, and conceptualization today, but it arguably 
concentrates on one of the most important AWF quality determinants: how acquired 
systems affect operational performance. 

In this exploratory study, we examine fast-changing IS acquisition from the 
perspective of warfare at the tactical edge, and we consider dynamic knowledge and 
performance measures to both complement and contrast with extant, engineering- 
oriented metrics used to specify and assess most acquired systems today. In 
particular, we examine TCPED and concentrate our attention on the exploitation 
process. Analysis enables us to delineate the key process activities, roles, 
relationships, and responsible organizations, and we leverage such delineation to 
assess, visualize, and measure the corresponding dynamic knowledge and 
performance. 
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Concentrating still further on the exploitation process activity Develop 
Adversary COA, we step through an example of how our approach can be used to 
assess dynamic knowledge and performance on the tactical edges of warfare 
organizations involved with TCPED. We focus especially on how such assessment 
can be employed to evaluate one or more IS being considered for acquisition and 
possible integration. 

This TCPED example includes delineation of four key knowledge flows 
associated with adversary COA development (i.e., military tactics knowledge 
development, intergroup knowledge accumulation, tacit knowledge sharing, explicit 
knowledge sharing) and rough estimation of the knowledge reach, flow time, and 
power corresponding to each. We take initial measurements for the process 
activities with no support from an IS under consideration. This becomes our baseline 
for comparison. Then we take measurements from the same process activities with 
support from one particular IS (e.g., IS-1), then from another (e.g., IS-2), to establish 
a measurement basis for comparison and evaluation. If acquisition evaluators are 
able to control the assessments sufficiently well (especially via laboratory or field 
experiment), then one can ascribe any performance differences solely to the IS. This 
ascription would help to bridge the capacious gap between acquisition efficiency and 
warfare efficacy on the tactical edges of organizations. 

In turn, we provide a practical illustration of the approach, through which we 
examine two competing prototype IS conceptualized for elaboration, and we 
elucidate the kinds of insights that can emerge through assessment, visualization, 
and measurement of dynamic knowledge flows. Specifically, we illustrate via 
example how one particular IS (i.e., IS-1, which is conceived to enhance tacit 
knowledge sharing via social media applications) seeks to enhance performance 
through emphasis on tacit knowledge flows and how a competing IS (i.e., IS-2, 
which is conceived to enhance explicit knowledge sharing via document 
collaboration applications) seeks to enhance performance instead through emphasis 
on explicit knowledge flows. We further extend our set of knowledge and 
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performance measures to induce the new dimension extended knowledge power, 
which combines knowledge reach and power together into a single metric. 


Results of our study illustrate how process performance can either improve or 
degrade when a particular IS is introduced into a process, and they articulate how 
the kinds of dynamic knowledge and performance measures discussed in this report 
can elucidate a critical aspect of IS for evaluation: how acquired systems affect 
operational performance. Indeed, through this practical illustration, we find one IS 
increasing process performance by 58%, whereas the competing IS effects a 60% 
performance decrease. 

Building upon these results, one can now apply the approach described in 
this report to any number of IS acquisitions and use end customer performance as 
an objective measure of IS efficacy. This application will require some venue for (at 
least partially controlled) experimentation (e.g., in the laboratory, via field 
experiments, phased or blocked implementation), but the potential benefit is huge. 
Moreover, in addition to using dynamic knowledge and process measurement as 
illustrated here for evaluation, one can leverage the same set of measures to specify 
IS in the conceptualization, design, and development phases. Essentially, end 
customer performance becomes an objective design consideration through this 
revolutionary approach. 

In terms of measuring AWF quality, this research establishes stronger and 
more direct linkages between what acquisition personnel know (especially focused 
internally on acquisition organizations and processes) and what warriors on the 
tactical edges of organizations need (especially IS that improve warfare efficacy), 
and it provides a set of dynamic knowledge and performance measures that can be 
used to bridge the gap between acquisition efficiency and warfare efficacy. This 
measurement step alone offers potential to improve the effectiveness of those 
acquisition people and organizations that implement the approach described in this 
report; hence, one new measure of AWF quality emerges directly: use of dynamic 
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knowledge and process measures to assess end customer efficacy. In other words, 
the working hypothesis is that those acquisition people and organizations that use 
this approach will be more effective than those that do not; hence, simply assessing 
the extent to which this approach is used may become an important, complementary 
measure of AWF quality. 

Further, results from this research suggest that personnel in the AWF may 
benefit from increased understanding of the end customers for whom they acquire 
information and other systems. The acquisition system as a whole provides program 
offices, liaisons, needs determination and justification steps, milestone and oversight 
authorities, operational testing and evaluation, and myriad other steps seeking to 
represent end customers. Nonetheless, there may be no substitute for acquisition 
personnel who understand their customers in considerable detail. 

These results do not suggest that procurement clerks should be outfitted with 
helmets, rifles, and boots and then sent to the tactical edges of warfare 
organizations, or that warriors on such tactical edges should be given procurement 
assignments; rather, it suggests that by examining the key warfare processes 
performed at the tactical edges—and in particular, understanding the most important 
dynamic knowledge and performance characteristics of such processes—even 
procurement clerks in offices half a world away may gain important insight into their 
end customers—insight that may lead to improved workforce quality and that can be 
measured. 

As with any investigation, the exploratory research described in this report 
has limitations. Indeed, its very exploratory nature limits the magnitude and scope of 
what can be accomplished through a single study or articulated through a single 
technical report, and the relatively high level of ourTCPED examination lacks much 
of the detail that a more ambitious follow-on study could provide. The nature of our 
practical illustration represents another limitation, because we use hypothetical IS 
and representative performance estimates to elaborate the approach; hence, a more 
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detailed examination of one or more operational processes in the field, along with 
evaluation of one or more implemented prototype IS for comparison, would provide 
powerful information regarding the relative performance of such IS. This represents 
another promising topic for follow-up research. 

We also note the potential issues with comparing knowledge power across 
dramatically different knowledge flow processes and the need to understand our set 
of measures more clearly. In our practical illustration, for instance, we examine two 
knowledge flow processes associated with the same TCPED exploitation activity and 
focus on the same dynamics: knowledge sharing in support of adversary COA 
development. In this case, both competing IS (i.e., based on social media 
applications versus document collaboration applications) could be viewed as either 
complementary or supplementary artifacts, but such artifact complementation or 
supplementation may not always be feasible in terms of practical application; hence, 
the limits of this approach require further investigation. 

Moreover, the dynamic knowledge and performance measures themselves 
(e.g., knowledge reach, flow time, power, extended power) require improved 
understanding. For instance, can each of these measures support quantification and 
analysis via ratio scales, as implied in this report, or will analysts have to settle for 
interval, ordinal, or possibly even nominal scales for evaluation? As another 
instance, can each of these measures be defined and operationalized objectively 
and without the need for interpretation, or will they be more situated and context- 
dependent? How will instruments to obtain such measurements be validated, and 
how reliable will such instruments be across different organization domains, process 
classes, IS types, and people taking measurements? There are other limitations, of 
course, but the results are sufficiently promising to merit future research along the 
lines of this investigation, particularly as noted in this section. 

There is much to be done, and in our present environment of intense budget 
austerity, the need to work now is evident. This kind of research requires support, 
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however. One cannot expect for warriors fighting at the tactical edges of combat 
organizations to conduct controlled experiments; likewise, most acquisition 
professionals have neither the time nor the expertise to do so. Instead, the work 
needed now requires professional researchers to do what they do best: conduct 
research to develop new knowledge that can be applied and leveraged to help the 
warrior and acquisition professional alike. 

This requires funding, of course—funding that many will find difficult to 
prioritize over funds intended for warfare, procurement, and similar pressing needs. 
But applied research along the lines of this investigation can exert enormous 
leverage. If even a small study such as this can develop a technique that improves 
an organization spending hundreds of billions of dollars a year by only a tiny bit— 
say, one one-hundredth of one percent—that amounts to tens of millions of dollars 
per year in savings! Given that small studies correspond to costs in the thousands— 
yet have the potential to save tens of millions—they appear to be compelling 
investments, even in tough economic and budgetary times. 
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